Human-Machine Dialogue for Multi-Modal Decision Support Systems

In this paper we describe some of the research on multi-modal dialogue being performed in the Interactive Systems group at the Navy Center for Applied Research in Artificial Intelligence at the Naval Research Laboratory. Our concern with multi-modal interfaces arises from our past work in developing general-purpose natural language interfaces that can be applied to a variety of decision support systems [Perzanowski and Potter, 1990, 1991; Marsh 1991; Gurney, Marsh, and Wauchope 1993; Everett, Wauchope and Perzanowski, 1992, 1993; Wauchope, 1992, 1994]. In the next two sections, this paper presents two related pieces of work by our group. In Section 2.0 (Marsh & Gurney), we describe our basic research in human-machine dialogue for multi-modal interfaces in which we apply discourse modeling to graphical user interfaces (GUIs) to promote multi-modal interactions. In section 3.0 (Wauchope), we discuss an exploratory development effort concentrating on the integration of graphics and natural language with some discourse capability in a fully implemented, multimodal/multi-media (M4) interface to a simulation-based tactical combat training tool. After presenting these two approaches, we will conclude with some final remarks.